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PREFACE 


This  paper  is  a  summary  of  the  Air  Force  Research  Laboratory’s  Fundamental  Skills  Training 
(FST)  Project.  It  is  designed  only  to  be  an  overview  of  the  project.  Many  other  papers 
containing  details  of  the  technical  effort  have  been  written  and  published. 

Over  the  ten  years  of  the  project,  many  individuals  have  contributed  to  the  project.  The 
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THE  FUNDAMENTAL  SKILLS  TRAINING  PROJECT 


1  GOAL  OF  THE  FUNDAMENTAL  SKILLS  TRAINING  PROJECT 

From  1990  into  2000,  the  Air  Force  Research  Laboratory  (AFRL),  with  strong  support  from  the 
other  AFRL  research  sites- --Phillips  (NM),  Rome  (NY),  and  Wright  (OH)--as  well  as  the 
University  of  Texas  at  San  Antonio  (UTSA),  had  been  engaged  in  a  long-term  research  project. 
This  project  brought  state-of-the-art  intelligent  tutoring  technology  to  bear  on  our  nation’s 
growing  literacy  skills  problem  in  areas  such  as  mathematics,  writing,  and  science. 

The  primary  goals  of  the  Fundamental  Skills  Training  (FST)  Project  were  to  design  and 
develop,  implement,  evaluate,  and  transfer  intelligent  tutoring  systems  (ITSs)  to  participating 
public  schools  and,  when  appropriate,  to  industry  under  federal  technology  transfer  guidelines. 
ITSs  are  automated  training  systems  that  deliver  individualized  instruction.  These  systems  are 
possible  through  application  of  artificial  intelligence  principles  to  computer-based  training 
(CBT)  and  education.  Recent  ITS  research  in  training  and  education  demonstrated  a  level  of 
maturity  that  made  this  technology  transfer  feasible  and  worthwhile.  In  the  FST  project,  tutors 
were  built  to  supplement  the  work  of  the  teacher  in  the  classroom.  The  intelligent  tutoring 
systems,  therefore,  were  not  intended  to  replace  teachers,  but  rather  to  help  teachers  cope  with 
the  demands  of  teaching  an  increasingly  challenging  student  population. 

The  goal  of  this  report  is  to  summarize  AFRL’s  ten-year  FST  Project. 

2  INTELLIGENT  TUTORING  SYSTEMS 

The  following  tutoring  systems  were  developed  and  evaluated  under  the  FST  Project.  Of  these, 
the  Information  Technology  Tutor  is  not  an  intelligent  tutoring  system,  but  rather  follows  a 
traditional  CBT  system  design. 

2.1  MTutor 

The  first  iteration  of  the  word  problemsolving  tutor,  known  as  MTutor,  was  implemented  at  one 
high  school  in  San  Antonio,  TX,  in  September  1991.  The  tutor  taught  students  to: 

•  Define  the  goal  (specify  the  goal  of  the  problem,  known  values,  and  unknown  values), 

•  Represent  the  problem  (Build  a  graph,  spreadsheet,  or  diagram), 

•  Solve  the  problem  (Make  an  equation  and  answer  the  question),  and 

•  Reflect  (Take  notes  and  list  the  steps  used  to  solve  the  problem). 

In  the  pilot  evaluation,  eight  teachers  used  the  tutor  with  over  400  students  for  approximately  20 
class  periods  (50  minutes  each)  over  the  academic  year.  The  control  students  were  in  another 
demographically  similar  high  school. 
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2.2  The  Word  Problem  Solving  Tutor 


Based  on  the  evaluation  results  on  the  first  year,  the  FST  team  decided  to  design  and  develop  a 
second  iteration  based  on  the  lessons  learned  from  the  first  version  of  the  word  problem-solving 
(WPS)  tutor.  This  version  was  first  used  in  the  high  schools  in  the  Fall  of  1992  and  was  revised 
during  the  subsequent  years.  A  more  detailed  description  is  presented  in  Appendix  A. 

The  new  version  consists  of  23  independent  modules  of  instruction  that  correspond  to  critical 
curriculum  elements  in  beginning  algebra.  As  an  ITS,  it  differs  from  traditional  computer-based 
instructional  systems  in  that  it  uses  artificial  intelligence  to  carefully  and  automatically  tailor 
instruction  for  each  individual  student.  One  of  the  strengths  of  WPS  is  that  it  boosts  confidence 
and  mathematical  proficiency  by  using  dynamic,  self-paced,  adaptive  instruction  and  determines 
when  remedial  instruction  is  required,  providing  it  only  when  necessary.  Students  progress  from 
easy  problems  explained  in  the  computer-based  tutorial  to  difficult  problems  that  require  a 
greater  skill  level 

WPS  was  designed  to  enhance,  not  replace,  traditional  classroom  instruction.  During  the  course 
of  traditional  instruction,  classes  use  WPS  to  work  individually  and  receive  automated  feedback 
and  guidance  from  the  tutor.  This  frees  the  teacher  to  work  individually  with  students  who  are 
having  a  particularly  difficult  time  learning  the  basic  concepts. 

2.3  Reading  and  Writing  in  a  Supportive  Environment 

The  Reading  and  Writing  in  a  Supportive  Environment  (R-WISE)  software  was  the  first  writing 
skills  tutor.  R-WISE  was  a  classroom-based,  adaptive,  supportive  learning  environment  for 
strengthening  critical  thinking  skills  associated  with  several  writing  tasks.  The  writing  tasks 
implemented  in  R-WISE  were  based  on  the  work  of  cognitive  psychologists  who  studied  the 
writing  process  and  identified  a  number  of  common  strategies  employed  by  emergent  and  expert 
writers.  These  strategies  relate  to  pre- writing,  text  generation,  and  editing  activities.  R-WISE 
included  adaptive  computer-based  instruction,  generated  on  a  just-in-time  basis,  to  meet  the 
specific  needs  of  the  student  while  learning  these  strategies.  R-WISE  also  included  a 
composition  help  system. 

R-WISE  was  tested  and  evaluated  by  ninth- grade  English  classes  in  20  high  schools  in  five 
different  U.S.  States  over  the  period  1992-1995.  The  purpose  of  the  evaluations  was  to 
determine  whether  the  supportive  environment  of  R-WISE  could  help  students  improve  critical 
thinking  skills  related  to  writing.  Results  over  the  last  five  years  show  students  who  used  the 
writing  tutor  improved  their  overall  writing  skills  from  8%  to  26%. 
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2.4  Maestro:  The  Writing  Process  Tutor 

Maestro:  The  Writing  Process  Tutor  is  a  student -oriented  computer-based  program  designed  to 
improve  the  overall  writing  quality  and  analytical  reasoning  skills  of  writers.  Maestro’s  design  is 
based  on  the  cognitive  theory  behind  the  early  FST  writing  tutor,  R-WISE.  It  is  a  classroom- 
based,  adaptive,  supportive  learning  environment  for  strengthening  the  critical  thinking  skills 
associated  with  varying  writing  tasks.  Its  comprehensive  instructional  approach  provides  upfront 
instruction  in  the  writing  process  and  individualized  adaptive  advice  at  critical  points  in  each 
workspace.  Incorporating  motivational  features  that  address  individual  characteristics  such  as 
learning  styles  and  interests,  Maestro  presents  15  dynamic  interfaces,  over  100  tailored 
instruction  modules  (TIMs),  and  575  tutoring  advice  statements.  The  student’s  writing  process  is 
monitored  and  coached  by  an  intelligent  advice  system.  A  more  detailed  description  is  presented 
in  Appendix  B. 

This  tutor  supports  expository,  persuasive,  research  and  practical  writing  styles  with  over  100 
challenging,  multimedia  writing  assignments.  Maestro  offers  the  traditional  classroom  teacher 
full  control  over  these  assignments,  as  well  as  the  capability  of  entering  other  assignments  as 
deemed  necessary  by  the  nature  of  the  curriculum  or  the  individual  writing  student. 

2.5  Writing  Process  Tutor  -  Java  Version 

The  goal  of  the  Writing  Process  Tutor  -  Java  (WTPJ)  Version  was  to  design,  implement,  and 
evaluate  an  internet-based  version  of  Maestro:  The  Writing  Process  Tutor.  The  interfaces  and 
curriculum  of  Maestro  were  adapted  for  use  in  WPTJ.  While  the  two  writing  process  tutors  are 
similar  in  instructional  goals,  there  are  some  differences  between  the  two.  The  differences  result 
from  the  use  of  Java  for  the  Internet  version  and  Asymetrix’s  Toolbook  for  the  original  local  area 
network  (LAN)-based  system. 

2.6  Instruction  in  Scientific  Inquiry  Skills  Tutor 

Instruction  in  Scientific  Inquiry  Skills  (ISIS)  Tutor  uses  ecological  concepts  to  teach  scientific 
inquiry  skills.  It  uses  a  gaming  environment  as  a  motivational  feature  to  provide  tutoring  on  the 
difficult  task  of  teaching  scientific  methods  that  include  making  observations,  generating 
hypotheses,  designing  and  conducting  experiments,  drawing  conclusions,  and  accepting/rejecting 
hypotheses.  This  ITS  teaches  ecology  concepts  in  areas  including  biomes,  abiotic  factors  of 
plant  growth,  biotic  factors  in  ecosystems,  human  activities,  and  ecology  principles.  See 
Appendix  C  for  more  information  on  ISIS. 

Using  a  cognitive  apprenticeship  approach  to  teaching,  ISIS  implements  modeling  through  19 
Skill  Instructional  Modules  (SIMs),  with  over  40  Domain  Instructional  Modules  (DIMs)  on 
ecology  topics.  Through  scaffolding,  students  learn  and  perform  skills  incrementally  by 
addressing  one  of  over  300  research  questions.  In  a  simulated  environment,  students  conduct 
real-time  research  developing  their  scientific  skills.  A  gaming  character  provides  advice  and 
feedback.  The  advice  then  fades  as  the  student  becomes  more  proficient.  After  the  assignment, 
students  reflect  by  answering  questions  designed  to  apply  their  newly  acquired  knowledge. 
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2.7  Information  Technology  Tutor 

The  goal  of  the  Information  Technology  Tutor  (IT  Tutor)  is  to  teach  basic  computer- related  skills 
and  knowledge.  After  having  completed  the  IT  Tutor,  entry-level  students  should  be  prepared  to 
learn  more  from  subsequent  IT  instruction.  The  IT  Tutor  covers  basic  keyboard  layout  and 
functions,  mouse  use,  computer  hardware  components,  Microsoft  Windows  interface  and 
software,  the  Internet,  and  electronic  mail.  The  IT  Tutor  is  geared  toward  young  adults  who 
have  little  to  no  knowledge  or  experience  with  personal  desktop  computers.  Students  who  are 
familiar  with  today's  software  may  find  some  sections  helpful  while  other  sections  may  be  more 
of  a  review  of  what  they  already  know.  More  advanced  students  may  find  the  IT  Tutor  "too 
simple.”  Instructors  should  become  familiar  with  the  IT  Tutor  curriculum  and  assign  students  to 
use  it  on  an  "as  needed"  basis.  Staff  may  use  the  IT  Tutor  for  their  own  staff  development  too. 
The  IT  Tutor  presents  computer  concepts  in  straightforward,  simple,  and  direct  instruction.  The 
instruction  should  not  be  viewed  as  training  on  specific  applications,  but  rather  on  general 
concepts,  which  apply  to  different  kinds  of  software.  More  importantly,  the  IT  Tutor  is  designed 
to  complement,  but  not  replace  human- delivered  instruction.  That  is,  the  IT  Tutor  is  only  an 
introduction  to  computer  concepts  that  should  be  followed  by  more  in-depth  teaching  by 
instructors  in  the  classroom. 

The  initial  version  of  the  IT  Tutor  was  delivered  to  the  Department  of  Labor  in  July  2000. 
Subsequent  to  the  FST  Project,  an  enhanced  version  was  developed  and  delivered  in  March 
2001.  The  IT  Tutor  was  distributed  nation-wide  in  a  series  of  training  sessions  conducted  by  the 
National  Office  of  the  Job  Corps  later  that  year. 

3  PARTNERS 

In  the  prime  years  of  the  FST  Project,  the  Air  Force  Research  Laboratory  (formerly  Armstrong 
Laboratory)  was  strongly  supported  by  the  sibling  Air  Force  laboratories:  Wright  Laboratory 
(Dayton,  OH),  Rome  Laboratory  (Rome,  NY),  and  Phillips  Laboratory  (Albuquerque,  NM). 
These  laboratories  not  only  contributed  funding  for  the  design,  development,  and  evaluation  of 
the  FST  tutors,  but  they  also  sponsored  test  and  evaluation  sites  in  their  local  community.  In 
1992,  the  FST  Project  expanded  from  one  test  and  evaluation  site  to  nine.  Seven  were  under 
sponsorship  of  the  Air  Force  laboratories,  one  under  the  sponsorship  of  UTSA,  and  one  under  the 
sponsorship  of  Lehigh  University.  The  latter  two  sponsors  were  part  of  a  Cooperative  Research 
and  Development  Agreement  (CRDA)  signed  in  1991  (See  the  Technology  Transfer  section). 

In  1997,  the  Department  of  Defense  (DoD)  and  the  Department  of  Labor  Employment  and 
Training  Administration  (DOLETA)  began  a  cooperative  effort  to  transition  the  FST  products 
from  the  military  to  DOLETA  workforce  development  settings.  This  support  was  provided  to 
install  the  FST  tutors  in  DOLETA- sponsored  settings,  to  design  and  develop  the  Java  version  of 
the  writing  process  tutor,  and  to  design  and  develop  the  IT  Tutor.  Finally,  the  Northeast 
Independent  School  District  of  San  Antonio,  TX,  provided  the  Air  Force  with  a  premier  test 
facility,  Mac  Arthur  High  School,  beginning  in  September  1991. 
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4  USERS 


Over  the  ten  years  of  the  FST  project,  the  FST  team  attempted  to  implement  and  evaluate  the 
intelligent  tutoring  systems  in  a  variety  of  educational  settings  with  varying  degrees  of  success. 
As  mentioned.  Department  of  Labor  sites  were  added  to  the  FST  project  in  1997.  In  addition,  the 
FST  team’s  participation  in  a  DOLETA  project  lead  to  a  group  of  users  for  the  IT  Tutor.  The 
full  list  of  sites  involved  to  some  degree  in  the  FST  project  is  listed  in  Appendix  D. 

For  the  most  part,  the  high  schools  participating  in  the  project  under  support  of  AFRL  or  UTSA 
attended  training,  used  the  tutors  for  the  suggested  number  of  hours  per  jear,  and  completed  the 
pretest  and  posttest.  As  support  for  the  schools  participating  in  the  FST  Project  waned,  so  did 
the  schools’  completion  of  the  evaluation  requirements.  The  Department  of  Labor  sites, 
however,  did  not  exhibit  the  same  level  of  compliance  with  the  FST  requirements.  In  some 
cases,  instructors  were  trained  to  use  the  tutors,  were  excited  about  doing  so,  but  then  were  not 
able  to  actually  begin  using  the  tutor.  There  were  several  unforeseen  factors  influencing  whether 
students  used  the  tutors.  One  example  was  a  lack  of  functional  hardware.  In  one  setting,  there 
simply  were  not  enough  functional  computers  for  the  instructor  to  use  the  tutors  effectively. 
(These  factors  are  briefly  described  in  the  Implementation  Model  and  Timelines  sectioa) 

4.1  High  Schools 

As  mentioned  above,  the  FST  project  grew  in  the  number  of  schools  participating  in  the  project 
in  a  series  of  steps.  The  project  began  in  1991  with  one  high  school  and  expanded  to  nine 
schools  in  Fall  1992  in  New  York,  Pennsylvania,  Ohio,  New  Mexico,  and  Texas.  Under  funding 
from  Phillips  Laboratory,  an  additional  17  schools  in  New  Mexico  were  added  beginning  in  1995 
bringing  the  total  to  40  high  schools. 

4.2  Job  Corps  Centers,  Youth  Fair  Chance,  Career  Advancement  Centers 

The  FST  team  began  working  with  Department  of  Labor  sites  in  June  1997  at  a  training  session 
in  Baltimore,  MD.  The  sites  included  Job  Corps  Centers  (JCCs),  Career  Advancement  Centers 
(CACs),  and  Youth  Fair  Chance  (YFC)  sites.  Several  additional  JCCs  were  added  to  the  FST 
project  and  Labor’s  IT  Pilot  Project. 

5  TECHNICAL  AND  PROFESSIONAL  SUPPORT 

During  the  first  year  of  the  project,  the  FST  team  realized  that  the  project  would  entail  more  than 
simply  designing  and  developing  software.  Carefully  planned  and  executed  implementation  of 
the  software  was  necessary  so  that  empirically  sound  evaluations  could  be  conducted.  Not  only 
was  teacher  training  on  the  software  critical,  but  also  user  manuals  and  other  teacher-related 
documents  had  to  be  produced.  In  addition,  communication  with  the  sites  became  an  important 
activity.  The  following  are  examples  of  the  technical  and  professional  support  activities 
accomplished  by  the  FST  team  throughout  the  project. 
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5.1  Teacher  Training  Sessions 

Teacher  training  was  one  major  component  of  the  professional  support  provided  by  the  FST 
team.  The  FST  team  conducted  72  teacher- training  sessions  between  August  1991  and  January 
2000.  Teacher  training  sessions  were  most  often  held  in  August  or  September  prior  to  the  Fall 
semester  or  at  least  in  the  first  few  months  of  the  school  year.  In  some  cases,  such  as  for  the  Job 
Corps  instructors,  training  occurred  in  “off’  months  (e.g.,  January  2000).  Most  training  sessions 
were  held  at  high  schools  in  San  Antonio,  TX,  and  Albuquerque,  NM,  but  occasionally  training 
sessions  were  held  in  other  locations:  Dayton,  OH;  Allentown,  PA;  Rome,  NY;  Los  Lunas,  NM; 
Baltimore,  MD;  Bangor,  ME;  Montgomery,  AL;  and  Long  Beach,  CA. 

Teacher  training  sessions  adopted  a  consistent  pattern  over  the  years.  Each  session  covered  a 
short  overview  of  the  FST  project,  background  on  the  design  of  the  tutoring  system,  a 
demonstration  or  “walk-through”  of  the  student  portion  of  the  tutor,  several  hours  of  hands-on 
time,  training  on  the  teacher  utilities,  and  guidance  on  how  to  integrate  the  use  of  the  tutor  into 
the  teacher’s  existing  curriculum.  Training  on  the  word  problem-solving  tutor  lasted  one  day, 
science  1  Vi  to  2  days,  and  writing  1  Vi  to  2  days,  depending  on  the  size  of  the  audience. 

5.2  Training  Materials 

Training  materials  were  developed  and  maintained  throughout  the  life  of  the  FST  Project.  At  the 
beginning  of  a  teacher  training  session,  each  teacher  was  given  a  copy  of  the  training  materials, 
which  included  an  agenda,  a  point  of  contact  sheet,  descriptions  of  the  software,  teacher 
responsibilities,  evaluation  data  from  previous  school  years,  a  user  manual,  teacher  guides, 
teaching  tips,  lab  manuals,  and  forms. 

5.3  User  Manuals 

As  mentioned,  user  manuals  for  each  tutor  were  developed  and  updated  with  each  release  of  the 
software.  These  were  distributed  as  part  of  the  teacher  training  and  whenever  a  new  version  of 
the  tutoring  systems  was  distributed  to  the  schools.  The  user  manuals  covered  the  standard 
topics:  the  nature  of  the  tutoring  system,  how  to  use  it,  how  to  install  it,  and  how  to  use  the 
teacher  toolkits  (i.e.,  teacher  management  utilities). 

5.4  Teacher  Guides 

In  addition  to  the  usual  user  manuals,  the  FST  team  developed  a  teacher  guide  or  laboratory 
manual  for  each  tutoring  system.  These  differed  across  the  three  tutors  because  the  needs  of  the 
teachers  in  each  discipline  had  different  teaching  needs.  For  the  word  problem-solving  tutor,  a 
laboratory  manual  containing  a  listing  of  all  of  the  word  problems  used  within  the  tutor  was 
given  to  the  teachers.  This  listed  the  word  problem  text  and  a  correct  solution  path  for  that  word 
problem.  For  RWISE  and  Maestro,  a  teacher’s  guide  was  produced.  This  guide  contained 
activities  the  English  teachers  could  use  in  their  traditional  classroom  to  support  the  instruction 
included  in  the  tutor.  For  each  step  of  the  writing  process,  the  guide  contained  worksheets  that 
could  be  copied  and  distributed  to  the  students.  For  ISIS,  a  lab  manual  included  a  listing  of  the 
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research  assignments,  a  printout  of  the  glossary,  a  listing  of  the  tutor’s  independent  variables, 
their  units  of  measurement,  and  equipment  used  to  measure  the  variables.  Each  of  these  guides 
or  manuals  was  distributed  at  the  teacher  training  sessions  or  when  major  revisions  occurred. 

5.5  Communication  with  the  Schools 

As  mentioned,  the  FST  project  involved  more  than  installing  software  in  the  schools  and 
collecting  data.  Communication  with  each  school  was  crucial  especially  during  the  first  year  in 
working  with  a  school.  Several  “site  coordinator”  meetings  were  held  during  the  first  few  years 
of  the  project  to  transfer  information  to  the  sites,  give  status  reports  to  the  sites,  and  to  listen  to 
issues  and  problems  at  each  site.  Phillips  Laboratory  in  Albuquerque  sponsored  similar  meetings 
for  the  19  test  and  evaluation  sites  in  New  Mexico. 

As  technology  evolved  over  the  decade,  the  nature  of  the  communication  changed.  The  FST 
team  and  evaluation  site  personnel  communicated  through  the  standard  technologies  of 
telephone,  mail,  and  facsimile.  A  few  video  teleconferences  (VTCs)  were  held  with  teachers  and 
government  points  of  contacts  by  using  the  VTC  capabilities  at  the  local  sponsoring  Air  Force 
laboratory.  Almost  humorously,  some  schools  in  the  early  years  of  the  project  had  to  unplug 
their  single  phone  into  the  school  to  receive  a  fax  from  the  FST  team.  E-mail  was  not  heavily 
used  until  the  final  years  of  the  FST  Project,  because  schools  and  teachers  were  late  in  adopting 
the  use  of  e-mail. 

6  DEVELOPMENT  CYCLES  AND  TEAMS 

The  design  and  development  processes  and  timelines  were  very  consistent  over  the  life  cycle  of 
each  tutoring  system.  The  timeline  was  not  explicitly  planned,  but  resulted  due  to  the  constraints 
of  the  academic  calendar  of  the  test  and  evaluation  sites.  That  is,  the  FST  team  needed  to  release 
an  updated  version  of  the  software  and  conduct  training  near  the  beginning  of  each  school  year. 
The  development  of  the  software,  curriculum,  and  supporting  materials  had  to  be  completed  by 
early  to  mid- August  each  year. 

6.1  Sources  of  design  information 

Even  though  the  designs  of  the  various  tutors  were  conducted  by  different  design  teams,  the 
sources  of  information  were  consistent  across  the  tutors.  First,  researchers  pemsed  theoretical 
literature  in  each  domain.  For  instance,  writing  process  theories  were  reviewed  and  considered 
for  implementation.  Theories  and  models  from  cognitive  psychology,  educational  psychology, 
and  computer  science  were  also  reviewed.  Second,  researchers  reviewed  published  and  non- 
published  literature  for  empirical  research  studies  for  information,  which  might  facilitate  the 
design  of  the  tutor.  Finally,  “best  practices”  were  collected  from  master  teachers  participating  in 
the  design  of  the  tutors  and,  in  some  cases,  teachers  who  had  been  us  ing  previous  versions  of  the 
tutoring  systems.  The  information  gleaned  from  these  sources  was  then  synthesized  into  a 
coherent  design  for  each  tutor. 


7 


6.2  Timelines 


The  timelines  for  the  development,  implementation,  and  evaluation  of  each  tutor  was  constrained 
by  the  academic  calendar  of  the  participating  schools.  In  general,  the  initial  design  and  software 
development  for  each  tutor  took  one  year.  During  the  summer  months  prior  to  a  tutor’s  pilot  test, 
master  teachers  developed  the  curriculum,  teacher  guides,  and  contributed  to  the  writing  of  a  user 
manual. 

The  timelines  for  the  second  and  subsequent  years  of  each  tutor  was  fairly  consistent  across  the 
tutors.  The  following  list  of  events  and  months  is  typical: 

•  Teacher  training  was  conducted  in  August. 

•  A  new  tutor  version  was  tested  and  distributed  in  August  or  early  September. 

•  Students  completed  the  evaluation  pretests  in  September. 

•  Site  visits  and  teacher  surveys  or  interviews  were  conducted  from  late  Fall  through  mid- 
Spring. 

•  Master  teachers  developed  additional  or  enhanced  existing  curriculum  materials 
throughout  the  academic  year. 

•  Students  completed  the  posttests  in  April  or  May. 

•  The  evaluation  data  were  collected,  analyzed,  and  summarized  in  June  and  July. 

•  Tutor  design  changes,  curriculum  development,  and  software  changes  were  made  during 
the  summer  months  prior  to  the  new  release  in  August. 

6.3  Teams 

Each  of  the  intelligent  tutoring  systems  described  above  were  designed,  developed,  and 
evaluated  by  an  interdisciplinary  team.  Major  design  decisions  for  a  tutor  were  made  either  by 
the  entire  design  team  or  a  subset  with  at  least  one  member  from  each  of  the  disciplines  listed 
below. 

•  Master  teachers  wrote  curriculum  materials,  instructor  guides,  and  user  manuals.  They 
also  provided  tutor  instructional  design  guidance,  information  on  “best  teaching 
practices,”  and  professional  support,  such  as  teacher  training.  English,  math,  and  science 
teachers  not  only  contributed  to  the  tutors  in  their  own  discipline,  but  also  to  other  tutors. 
For  instance,  the  English  teachers  reviewed  the  curriculum  materials  of  all  three  tutors. 

In  addition,  one  of  the  science  teachers  wrote  word  problems  for  WPS.  Math  teachers 
reviewed  instruction  on  graphing  which  was  included  in  the  science  tutor. 

•  Computer  programmers  were  responsible  for  the  design  and  development  of  the  software. 
They  participated  in  all  design  meetings  even  if  the  focus  of  the  meeting  was  on  the 
curriculum  or  instruction  to  be  embedded  within  the  tutoring  system.  Additionally,  they 
provided  technical  support  for  the  sites  (e.g.,  during  installation)  and  wrote  installation 
documents,  support  documents  (e.g.,  frequently  asked  questions),  and  design  documents. 


•  Research  psychologists  reviewed  relevant  theoretical  and  empirical  research  literature 
leading  toward  the  design  of  the  tutors.  They  drafted  summaries  of  instructionally 
effective  design  principles,  wrote  user  manuals,  provided  professional  and  technical 
support  for  the  schools,  develop  and  administered  the  pretests  and  posttests,  analyzed  the 
evaluation  data,  wrote  technical  papers,  gave  presentations  to  professional  and 
management  organizations,  and  maintained  the  website. 

•  For  each  tutor,  a  team  lead  managed  and  directed  the  interdisciplinary  team.  Across  the 
life  of  the  project,  the  team  leads  came  from  different  professional  backgrounds.  For 
instance,  one  was  a  cognitive  psychologist,  one  was  a  technical  writing  instructor  at  an 
engineering  school,  one  was  an  educational  technology  specialist,  and  one  was  an 
educational  psychologist. 

•  Site  coordinators,  while  not  part  of  the  “core”  tutor  design  and  development  teams, 
played  an  important  role  in  the  implementation  and  evaluation  of  the  tutors.  Each  of  the 
evaluation  sites  were  asked  to  pick  an  individual  in  a  school,  preferably  a  teacher,  who 
would  be  the  prime  point  of  contact  for  the  FST  team.  These  individuals  were 
responsible  for  scheduling  and  maintaining  the  computer  labs,  coordinating  the 
evaluations,  distributing  and  collecting  surveys  of  the  teachers,  communicating  with  the 
FST  team,  and  keeping  their  school  administrators  informed  about  the  project  status. 

7  EVALUATION 

One  of  the  primary  goals  of  the  FST  project  was  to  assess  the  instructional  effectiveness  of  the 
implementation  of  the  tutoring  systems  in  public  education  settings.  From  1991  through  1998, 
the  FST  team  conducted  17  academic  year-long  evaluation  studies  of  the  instructional 
effectiveness  of  the  FST  tutors.  A  brief  summary  of  the  methodology  is  followed  by  short 
descriptions  of  selected  studies.  Appendix  E,  a  summary  of  the  17  evaluation  studies,  was 
written  for  educational  administrators. 

7.1  Methodology 

A  variety  of  evaluation  methodologies  were  used  over  the  course  of  the  project.  Some  studies 
included  control  groups  in  demographically  similar  high  schools,  whereas  others  included 
control  groups  within  the  same  high  school.  Several  studies  were  conducted  to  replicate  the 
initial  evaluations,  but  did  not  include  a  traditional  control  group. 

The  FST  pretests  and  posttests  required  more  of  the  students  than  traditional  multiple- choice 
tests  assessing  students’  knowledge.  The  subtle  distinction  is  the  difference  between  “knowing 
what”  and  “knowing  how.”  Each  test  was  designed  to  explore  students’  ability  to  perform  the 
skills  being  taught  in  the  tutors. 

•  WPS  was  evaluated  with  multiple- choice  tests  asking  questions  about  the  word  problem¬ 
solving  process,  such  as  “What  is  the  goal  of  the  word  problem?”  These  questions  went 
beyond  the  traditional  word  problem-solving  questions  in  which  the  student  must  simply 
choose  the  correct  answer  to  the  word  problem.  The  questions  in  the  WPS  tests  were 
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designed  to  assess  detailed  aspects  of  the  problem  solving  process,  not  just  the  final 
answer. 

•  RWISE  and  Maestro  evaluations  required  students  to  write  either  a  five-paragraph 
persuasive  or  expository  essay.  These  writing  samples  were  scored  by  independent  raters 
who  were  given  rubrics  (i.e.,  scoring  guidelines)  for  each  of  five  scales.  These  included  a 
holistic  score  and  several  analytical  scores:  abstraction,  organization,  development,  and 
purpose. 

•  ISIS  pre-  and  post-tests  were  multiple -choice  tests  covering  the  scientific  inquiry  skills 
and  ecology  content. 

In  the  early  days  of  the  project,  the  tests  were  given  in  a  paper- and-pencil  fashion.  FST  team 
members  served  as  proctors  in  each  classroom  during  the  pretests  and  posttests  to  ensure  the  tests 
were  given  correctly  and  the  students  completed  the  tests.  In  the  later  years  of  the  project,  during 
the  replication  studies,  the  test  was  given  in  a  computer-based  testing  format. 

7.2  Outcomes 

The  details  of  each  evaluation  study  methodology  and  results  are  beyond  the  scope  of  this  report. 
A  summary  of  the  results  is  listed  in  Appendix  E.  For  details  of  specific  studies,  the  reader 
should  read  the  relevant  papers  listed  in  Appendix  F.  Nonetheless,  one  example  study  for  each 
of  three  tutoring  systems  is  presented  here. 

WPS 

The  initial  evaluation  of  WPS  was  conducted  with  632  students  in  seven  high  schools  in  three 
states.  The  seven  schools  were  randomly  assigned  as  control  sites,  placebo-treatment  sites,  and 
treatment  sites.  At  the  control  sites,  students  exposed  to  traditional  instruction  enhanced  their 
word  problem-solving  performance  by  19%  over  the  course  of  a  school  year.  At  the  placebo- 
treatment  sites,  students  who  solved  these  problems  using  the  same  problem-solving  software  but 
with  the  cognitive  pedagogy  removed  enhanced  their  problem-solving  performance  by  19%.  At 
the  treatment  sites,  students  who  were  exposed  to  WPS  enhanced  their  word  problem-solving 
performance  by  29%  over  the  same  period.  In  comparison  to  traditional  instruction  of  word 
problem  solving,  the  placebo  treatment  had  no  effect  on  learning,  and  WPS  reliably  enhanced 
learning.  In  subsequent  year-long  evaluations,  the  results  supported  these  findings. 

Maestro 

Maestro  was  designed  as  a  follow-on  tutor  to  R-WISE,  based  on  results  of  the  R-WISE  studies, 
teacher  input,  cognitive  research  into  the  writing  process,  and  the  cognitive  apprenticeship 
instructional  strategy.  The  functionality  and  efficacy  of  the  initial  version  of  Maestro  was  tested 
in  a  large-scale  pilot  study.  The  study  groups  include  classes  of  54  teachers  at  23  schools. 

Similar  to  results  with  the  other  tutors  in  the  FST  suite,  student  gains  were  directly  influenced  by 
the  amount  of  time  spent  on  the  Writing  Process  Tutor.  Specifically,  students  spending  at  least 
11  hours  using  Maestro  improved  by  11%  while  the  control  group  improved  by  only  3%. 
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ISIS 

A  large-scale  test  of  the  effectiveness  and  implementation  of  ISIS  was  conducted  in  14  schools 
in  five  states.  There  were  over  30  teachers  using  ISIS  in  83  sections  of  ninth-grade  biology. 
There  were  46  sections  in  the  same  high  schools  serving  as  non-treatment  control  groups.  The 
FST  team  also  investigated  the  feasibility  of  using  ISIS  in  seventh- grade  honors  biology  and 
explored  several  research  questions  concerning  the  use  of  concept  mapping  in  a  knowledge -rich 
domain.  In  terms  of  overall  gains,  the  treatment  group  improved  by  8%,  while  the  control  group 
improved  by  only  4%.  Specifically,  the  skill  level  obtained  by  the  students  directly  influenced 
the  amount  of  gain.  Students  who  used  the  tutor  more  gained  more  than  the  control  group  or 
students  who  used  the  tutor  for  only  a  few  hours. 


8  IMPLEMENTATION  MODEL  AND  TIMELINES 

One  goal  of  the  FST  Project  was  to  implement  the  tutoring  systems  in  educational  settings.  The 
phrase  “Go  work  with  public  education”  was  one  of  the  initiating  directions  for  the  project.  As 
mentioned  in  the  discussion  of  the  users,  the  FST  Project  focused  its  efforts  on  40  high  schools 
supported  by  AFRL  or  UTSA  funding.  Implementing  the  tutoring  systems  became  relatively 
routine  by  the  third  or  fourth  year  of  the  project.  In  the  later  years,  the  FST  project  expanded  to 
Job  Corps  Centers,  Youth  Fair  Chance  Centers,  and  Career  Advancement  Centers  under  funding 
from  the  Department  of  Labor  Employment  and  Training  Administration  (DOLETA).  As  a 
result,  the  FST  Project  team  began  to  experience  users  who  did  not  fit  the  traditional  public 
education  model.  Things  didn’t  always  go  as  planned.  As  an  example,  the  FST  staff  trained 
instructors  from  several  Job  Corps  Centers.  In  some  cases,  the  instructors  returned  and  used  the 
tutors,  but  in  other  cases,  the  instructors  did  not.  Because  of  the  experiences  with  the  high 
schools  and  DOLETA  sites,  the  FST  team  developed  two  models  concerning  educational 
technology  projects  and  the  activities  needed  to  support  new  and  existing  users. 

8.1  Implementation  Model 

The  implementation  model  posits  factors  that  affect  student- learning  outcomes  during  the  use  of 
technology  in  education  and  training  settings  (see  Figure  1).  The  full  implementation  model 
includes: 

•  Teacher  variables  such  as  workload,  training,  motivation; 

•  Technology  variables  including  technical  support,  hardware,  software; 

•  Student  variables  such  as  motivation,  access,  history; 

•  Administrators  and  external  requirements,  such  as  system  mandated  curriculum; 

•  Project  staff;  and 

•  Learning  outcomes  (i.e.,  what  students  learn).  (This  is  affected  by  the  quantity  of  time 
students  interact  with  the  software  and  the  quality  of  that  interaction.). 

The  FST  team  found  that  a  deficiency  in  one  or  more  of  these  factors  could  negatively  impact  the 
students’  learning  outcomes. 
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Project  Staff 

•  The  project  staff  should  be  an  interdisciplinary  team  of  administrators,  expert  teachers, 
educational/instructional  psychologists,  computer  programmers,  and  researchers. 
Decision-making  throughout  the  life  of  the  project  should  take  into  account  each  of  these 
perspectives. 

•  Project  staff  influence  the  quality  of  the  training  teachers  receive. 

•  The  project  staffs’  implicit  or  explicit  theories/models  of  education  influence  the  design 
of  the  software.  The  quality  of  the  software  in  turn  impacts  the  quality  of  the  students’ 
interaction  with  the  domain  information  and  the  students’  motivation  to  use  the  software. 

Administrative  Factors 

•  Administrators  affect  the  quantity  of  time  students  interact  with  domain  concepts  via  the 
software  by  their  decisions  in  setting  up  and  maintaining  the  computer  facilities  and  by 
hiring  (or  not  hiring)  technical  support.  If  the  facilities  (including  hardware  and 
software)  are  not  maintained,  student  access  time  drops. 

•  Administrators  not  only  play  a  role  in  providing  opportunities  for  learning  by  providing 
adequate  facilities,  but  they  also  influence  students’  access  to  the  content  by  influencing 
teachers’  workload.  If  teachers  are  burdened  with  non- technological  duties,  they  will  not 
have  the  time  to  learn  and  implement  the  technology  appropriately. 

•  Administrators  also  affect  teachers’  motivation  by  giving  incentives,  stipends,  workload 
reduction,  or  by  placing  requirements  on  them  to  use  the  hardware  and  software. 
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Teacher  Factors 

•  Teachers’  motivation  to  use  learning  technology  is  directly  impacted  by  their 
administrators,  training  on  the  hardware  and  software,  and  external  requirements. 

External  requirements  may  include  state- mandated  testing,  grant,  and/or  contract 
requirements. 

•  Teachers’  skills  are  influenced  by  their  educational  background  and  history  with  the 
domain  and  technology.  Their  skills  are  also  influenced  by  the  quality  of  training  they 
receive  on  the  technology,  and  the  time  they  have  to  use  the  software  after  training. 

These,  in  turn,  affect  how  they  interact  with  the  students  and  the  software. 

Technology  Factors 

•  The  quality  of  the  software  affects  whether  students  are  motivated  to  use  the  software  and 
the  quality  of  the  interaction  between  students  and  the  domain  information. 

Student  Factors 

•  Students'  motivation  to  learn  the  domain  information  is  affected  by  the  quality  of  the 
software  and  by  their  own  personal  goals.  If  they  are  not  interested  in  learning,  their  time 
on  task  drops. 

Implications  of  the  Model 

The  model  can  be  used  in  planning  and  managing  a  project  that  implements  educational 
technology.  The  model  may  also  be  used  in  a  wide  variety  of  educational  settings,  such  as  public 
or  private  schools,  and  adult  education.  Guidelines  for  project  staff,  administrators,  and  teachers 
can  be  derived  from  the  model. 

8.2  Timelines 

Twenty-one  events  described  the  life  span  of  an  instructional  technology  project.  The  events  are 
broadly  categorized  under  four  activities:  Project  Start-Up,  Site  Start-up,  Long-Term  Use,  and 
Project  End. 

Project  Start-Up 

•  Project  Start-up  refers  to  the  first  few  months  of  a  project  during  which  decision-making 
and  planning  determine  the  project's  goals,  team  members,  curriculum  and  software 
design,  target  population,  target  teachers,  and  funding  strategy. 

•  Project  Start-up  incorporates  such  steps  as  project  plan  approval,  software/curriculum 
development,  and  initial  contact  with  the  potential  user. 

Site  Start-up 

•  Site  Start-up  refers  to  the  events  in  which  the  project  staff  begins  to  work  with  the  target 
users. 

•  Site  Start-up  deals  with  administrator  training,  teacher  training,  and  the  initial  use  by 
teachers. 

Long-Term  Use 

•  Long-term  Use  refers  to  the  events  that  occur  during  the  "stable"  portion  of  a  project. 

This  usually  begins  in  the  second  or  third  year  of  a  project. 
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•  Long-term  Use  includes  site  visits  by  the  project  staff,  site  coordinators'  meetings,  and 
ways  to  assist  the  teachers  in  using  the  technology  after  the  initial  projected  has  ended. 

Project  End 

•  Project  End  concerns  the  preparation  of  sites  to  use  the  technology  as  part  of  their 
normal/routine  activity  without  project  staff  support. 

Notes  About  The  Timeline 

•  The  timeline  is  descriptive,  not  prescriptive.  Not  all  events  will  occur  within  all  projects. 

•  Some  steps  are  discrete  (i.e.,  short  duration  in  time)  while  others  are  continuous  events 
(i.e.,  happen  over  a  long  period  of  time). 

•  Some  events  occur  in  a  natural  order  (Idea  inception  "initiates"  the  project)  while  other 
events  occur  at  various  times  along  the  timeline  with  multiple  occurrences  being  possible. 

•  The  importance  of  events  within  a  project  varies.  For  instance,  teacher  training  is  much 
more  important  than  publicity  events. 

9  PRESENTATIONS  AND  PAPERS 

The  FST  team  literally  gave  hundreds  of  presentations  and/or  demonstrations  to  federal,  state, 
and  local  government  employees;  faculty  and  students  in  high  schools  and  universities; 
professional  organizations,  and  many  other  distinguished  visitors. 

Project  overviews  and/or  demonstrations  of  the  tutors  were  given  to  the  following  dignitaries: 

•  Then  President  William  Clinton 

•  Then  Texas  Governor  George  W.  Bush 

•  U.S.  Senator  Kay  Bailey  Hutchinson  Staff 

•  U.S.  Representative  Ciro  Rodriguez  and  Staff 

•  State  Senator  Joseph  Funa 

•  San  Antonio  Mayor  Nelson  Wolff 

•  General  Yates  (AFSC/CC) 

•  General  Richard  Paul  (AFRF/CC) 

•  Dr  Joseph  Osterman  (OSD) 

•  Andrew  Hartman  (National  Institute  for  Fiteracy) 

•  Mary  Silva  (Director  of  the  National  Office  of  Job  Corps) 

•  Russell  Kile  (Vice  President’s  National  Partnership  for  Reinventing  Government) 

•  Donald  Johnson  (OUSD) 

•  The  National  Board  of  the  American  Society  for  Training  and  Development 

By  no  means  is  this  a  complete  list.  To  its  credit  and  distraction,  the  FST  Project  was  a  highly 
visible  project  within  the  Air  Force.  FST  personnel  were  requested  to  give  numerous 
demonstrations  and  presentations,  as  were  many  of  its  sibling  projects.  In  one  5-month  period 
(January  -  May  1995),  the  FST  team  gave  110  presentations  and/or  demonstrations  of  the  tutors! 

As  part  of  its  research  mission,  the  FST  team  wrote  research  papers,  which  were  presented  at 
professional  conferences.  These  papers  and  presentations  are  listed  in  Appendix  F. 
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10  AWARDS 


The  FST  Project  Team  won  several  awards  during  its  ten- year  run.  These  include: 

Hammer  Award  from  Vice  President  A1  Gore  (Feb  1999).  The  FST  Team  received  Vice 
President  Gore’s  prestigious  Hammer  Award,  which  recognized  organizations  and  agencies  that 
streamlined  government  operations  and  improved  how  they  delivered  services.  Specifically  the 
award  recognized  the  work  on  WPTJ,  an  intelligent  tutoring  system  designed  to  deliver  writing 
process  instruction  over  the  Internet. 

The  Department  of  Labor's  Certificate  of  Excellence  (1996).  Robert  Reich,  then  Secretary  of 
the  Department  of  Labor,  presented  a  Certificate  of  Excellence  to  the  FST  team  for  its  work  with 
the  Department  of  Labor  service  providers. 

Armstrong  Laboratory,  Human  Resources  Directorate  Customer  Support  Excellence 
Award  for  1996.  The  Customer  Support  Excellence  Award  was  given  to  the  FST  team  for  the 
support  given  to  teachers  and  administrators  in  over  50  high  schools  and  workforce  development 
settings. 

Federal  Laboratory  Consortium  Award  for  Excellence  in  Technology  Transfer  for  1995. 

The  Federal  Laboratory  Consortium  for  Technology  Transfer  presented  an  award  to  the  FST 
team  for  its  work  in  transferring  the  word  problem-solving  tutor  to  a  commercial  partner.  This 
was  one  of  the  first  transfers  of  educational  technology  outside  of  the  Air  Force. 

The  Alamo  Federal  Executive  Board  1994  Quality  Award.  The  Alamo  Federal  Executive 
Board  awarded  the  FST  team  the  1994  Quality  Award  for  maintaining  high  standards  in  research 
and  working  with  customers. 

Champion  of  Education  Award  (1992).  The  San  Antonio  Independent  School  District 
presented  the  FST  team  their  "Champion  of  Education  Award"  for  the  team's  work  with  Sam 
Houston  High  School  in  San  Antonio,  TX 

The  Lone  Star  Salute  to  Community  Service  (1992).  The  FST  team  was  designated  as  a  1992 
Star  of  Texas  by  the  Texas  Community  Education  Association  for  its  innovative  efforts  at  Mac 
Arthur  High  School  in  the  North  East  Independent  School  District. 

11  TECHNOLOGY  TRANSFER 

The  fourth  goal  of  the  FST  project  was  to  transfer  the  technology  out  of  the  federal  laboratory 
setting  to  commercial  partners.  This  also  was  accomplished  to  varying  degrees  of  success.  The 
following  is  a  quick  summary  of  the  history  of  the  first  FST  CRDA. 

•  Cooperative  Research  and  Development  Agreement 

•  Signed  29  April  91 

•  Signatories  included  Armstrong  Laboratory,  The  University  of  Texas  at  San  Antonio, 
SAGE  Educational  Systems,  and  Lehigh  University 
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Modification  1  to  the  CRDA 

•  Date  signed  is  unknown 

•  Goal:  To  terminate  rights  of  SAGE  Educational  Systems  and  Lehigh  University 

Modification  2  to  the  CRDA  and  Appendix  E 

•  Signed  13  Sep  94 

•  Signatories  include  Armstrong  Laboratory,  The  University  of  Texas  at  San  Antonio, 
and  PWS  Publishing,  Inc. 

•  Goal:  To  license  the  Word  Problem  Solving  Tutor  (WPS)  to  PWS 

Modification  3  to  the  CRDA 

•  Signed  Feb  97 

•  Signatories  include  Armstrong  Laboratory,  The  University  of  Texas  at  San  Antonio, 
and  Brooks/Cole  Publishing 

•  Goal:  To  change  the  licensee  from  WPS  to  Brooks/Cole  Publishing. 

Modification  4  to  the  CRDA 

•  Signed  by  UTSA  and  Armstrong  Laboratory,  but  not  by  Brooks/Cole. 

Letter  of  Termination 

•  Signed  by  Brooks/Cole  on  7  Jul  99 


APPENDIX  A:  THE  WORD  PROBLEM  SOLVING  TUTOR 


Goal  of  Tutor 

The  goal  of  the  Word  Problem  Solving  Tutor  is  to  teach  students  have  to  analyze  word  problems 
and  generate  correct  solutions  in  a  variety  of  mathematical  areas. 

Curriculum 

Skills 

WPS  is  an  intelligent  tutoring  system  that  teaches  students  to  analyze  and  solve  word  problems 
in  five  steps: 

1.  Identify  the  goal  of  the  problem 

2.  Identify  the  values  necessary  to  solve  the  problem 

3.  Make  an  equation 

4.  Solve  the  equation 

5.  Answer  the  question  using  the  appropriate  solution  value  and  units  of  measure. 

The  Tutor  does  not  teach  mathematical  calculation  but  is  designed  to  be  used  as  a  supplement  to 
traditional  classroom  instruction  that  includes  mathematical  calculation. 

Math  Modules 

The  WPS  Tutor  consists  of  23  independent  modules  covering  pre- algebra,  algebra,  and 
geometry.  Each  of  the  23  modules  has  a  review  module. 


Algebraic  Equations 

Integer  Math 

Equations  In  Geometry 

Number  Sequences 

Pythagorean  Theorem 

Decimals 

Area  Of  Circles 

Percentages 

Formulas 

Proportions 

Ratio 

Area  Of  Triangles 

Area  Of  Quadrilaterals 

Circumference  Of  Circles 

Fractions 

Perimeters 

Area  Of  Shapes 

Volumes  Of  Solids 

Surface  Areas  Of  Solids  (5  Modules) 

Intended  Audience 

WPS  is  appropriate  for  middle  and  high  school  students  along  with  adults  pursuing  a  GED. 

Instructional  Features 

Tutorials.  Each  module  begins  with  a  computer-based  tutorial  that  uses  animation,  graphics,  and 
text  to  review  the  appropriate  subject  material.  In  addition  to  this  review,  the  tutorial  contains  a 
worked  example  of  a  sample  problem,  questions  to  test  students'  knowledge  of  the  material,  and 
a  summary  page. 
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Levels  of  Difficulty.  After  successful  completion  of  the  tutorial,  students  are  presented  with  a  set 
of  word  problems  organized  by  level  of  difficulty.  The  problems  in  the  first  level  of  difficulty 
are  isomorphic  to  the  example  problem  in  the  tutorial.  This  equation  is  also  the  simplest  version 
of  the  equation  for  that  module.  Each  subsequent  level  is  only  slightly  more  difficult  than  the 
previous  level,  and  the  source  of  additional  difficulty  is  explicitly  taught  in  the  Tutor. 

Help.  The  WPS  software  provides  several  levels  of  help  for  the  student.  For  example,  students 
may  access  help  in  the  form  of  hints  for  the  word  problem  they  are  working  to  solve.  Other  help 
includes  interface  help,  a  quick  tour  of  WPS,  and  a  review  the  units  of  measure  module. 

Tools.  As  students  seek  to  complete  the  five  problemsolving  steps,  the  interface  design  of  WPS 
allows  the  student  to  readily  access  a  myriad  of  assistive  tools  and  tables.  For  each  problem, 
students  have  access  to  a 

•  weights  and  measurement  table, 

•  formulas  table, 

•  glossary, 

•  notebook, 

•  lesson  summary,  and 

•  graphic  that  goes  with  the  word  problem. 

Interfaces 

WPS  has  one  primary  interface  in  which  the  students  perform  the  word  problem  solving  skills. 
Students  click  on  the  goal  statement,  identify  relevant  variables  and  their  values,  generate  and 
equation,  and  then  answer  the  question.  The  row  of  buttons  along  the  top  is  used  by  the  students 
to  access  various  useful  tools. 

Theoretical/Development  Background 

The  WPS  Tutor  is  pedagogically  based  on  five  cognitive  theoretical  foundations,  including 
learning  by  practice  (Anderson  &  Fincham,  1994;  Blessing  &  Anderson,  1996;  Fovett  & 
Anderson,  1994),  elaboration  (Reigeluth,  1987,  1992),  categorization  (Rosch,  1978),  mastery 
(Kulik,  Kulik,  &  Bangert-Drowns,  1985;  Slavin,  1990),  and  induction  (Reed  &  Bolstad,  1991; 
Reed,  Willis,  &  Guarino,  1994;  Ward  &  Sweller,  1990). 

Practice.  Theory:  Beaming  and  skill  acquisition  occur  when  students  are  taught  the  required 
declarative  knowledge  and  then  are  required  to  actively  process  the  information  or  practice  the 
skill. 

WPS:  Students  learn  declarative  knowledge  in  the  subject  (e.g.,  area  of  shapes)  and  pass  a  test  on 
that  knowledge  before  they  are  allowed  to  practice  solving  problems  in  that  domain. 

Elaboration.  Theory:  Students  begin  by  practicing  the  epitome  of  a  skill,  and  then  practice 
progressively  and  systematically  more  complex  versions  of  the  skill. 
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WPS:  Students  are  first  presented  with  the  simplest  possible  version  of  the  relevant  formula.  As 
students  progress  in  the  Tutor,  steps  are  added  to  the  formula,  making  it  more  difficult  and 
complex. 

Categorization.  Theory:  An  appropriate  way  to  organize  the  subject  information  is  specified  and 
then  explicitly  taught  to  students.  Information  is  organized  around  prototype  concepts,  which 
become  the  center  for  conceptual  categories. 

WPS:  Students  are  presented  with  a  prototype  formula  in  each  module.  As  students  practice 
their  skills,  they  build  expertise  and  (ideally)  recognize  that  new  problems  require  solution 
strategies  that  are  minor  variations  on  known  strategies. 

Mastery.  Theory:  When  a  complex  skill  is  to  be  taught  based  on  important  prerequisite  skills, 
the  curriculum  must  be  ordered  appropriately.  In  addition,  an  appropriate  level  of  mastery  for 
each  skill  is  insured  before  advancement  is  allowed. 

WPS:  Students  are  allowed  to  progress  to  a  new  level  or  module  only  when  they  have  mastered 
the  current  level  or  module.  The  instructor  can  set  mastery  requirements. 

Induction.  Theory:  Students  are  shown  how  to  solve  examples  of  the  kinds  of  problems  they  will 
encounter.  Student  can  solve  new  isomorphic  problems  by  mapping  steps  from  the  worked 
example  onto  the  new  problem. 

WPS:  Each  module  of  the  WPS  Tutor  presents  worked  examples  of  word  problems.  Then 
students  are  given  isomorphic  problems  to  solve. 

Evaluation 

WPS  has  been  tested  since  1992  in  some  of  the  largest  field  studies  of  educational  technology 
ever  conducted.  Students  at  secondary  and  middle  schools  in  Ohio,  Texas,  New  Mexico,  New 
York,  and  Pennsylvania  participated  in  these  studies.  In  one  study,  students  using  the 
mathematical  tutor  improved  their  word  problem  solving  skills  by  29%. 

Teacher  Toolkit  Features 

The  WPS  software  allows  the  instructors  to  create  rosters,  generate  reports,  choose  the 
instructional  mode,  and  set  the  class  curriculum. 

Rosters.  Teachers  may  enter  rosters,  display  rosters,  and  add,  delete,  or  update  student  records. 

In  addition,  users  may  sort  students  by  ID  number,  name,  or  by  class. 

Reports.  Teachers  may  generate  and  print  student  reports  for  specific  classes  or  for  all  students 
and  may  be  created  for  specific  dates.  Instructors  have  the  option  to  create  student  reports  or 
student  notebook  reports.  The  student  report  includes  the  amount  of  time  spent  on  the  problem, 
the  date  the  problem  was  completed,  the  type  of  problem,  the  problem  ID  number,  the  level  of 
the  problem,  the  number  of  steps  the  student  took  to  complete  the  problem,  the  number  of  errors 
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committed,  and  the  number  of  times  the  student  asked  for  help.  The  student  notebook  report 
includes  the  notes,  values,  equations,  and  answers  the  student  has  entered. 

Curriculum.  Instructors  may  set  the  curriculum  (see  Math  Modules  above)  for  individual 
students  as  well  as  for  entire  classes.  Setting  the  curriculum  allows  the  teacher  to  determine  the 
order  that  the  students  will  see  the  modules. 

Instructional  Mode.  Instructors  have  the  option  to  set  the  instructional  mode  that  the  student  will 
work  under.  WPS  has  a  "guided  mode"  and  a  "not  guided  mode.”  When  the  teacher  has  chosen 
to  have  the  students  work  under  the  guided  mode,  the  tutor  will  give  the  student  advice  each  time 
he/she  makes  an  error.  In  addition,  the  instructional  mode  requires  the  proper  completion  of 
each  step  before  the  student  is  allowed  to  proceed.  If  the  instructor  chooses  to  have  students 
work  problems  in  the  not  guided  mode,  they  will  be  allowed  to  complete  steps  in  any  order.  The 
software  will  only  give  advice  at  the  students'  request. 

Teachers  can  also  set  the  parameters  WPS  uses  to  make  decisions  about  the  student.  For 
example,  based  on  their  knowledge  of  students'  skills  and  abilities,  teachers  can  set  the  number 
of  problems  students  are  required  to  solve  per  module,  the  number  of  levels  of  difficulty  they  are 
required  to  achieve  per  module,  and  the  number  of  times  they  are  allowed  to  ask  for  help  per 
problem.  By  giving  the  teacher  the  option  of  changing  the  number  of  times  a  student  may  ask 
help  the  software  allows  the  teacher  to  can  ensure  that  the  student  progression  is  due  to  learning 
and  not  abuse  of  the  help  system.  If  a  student  requests  help  too  many  times,  he/she  will  not  get 
credit  for  the  assignment  and  will  be  asked  to  repeat  it  without  asking  for  so  much  help. 
Instructors  may  also  choose  to  allow  the  student's  performance,  as  measured  by  the  software,  to 
guide  the  student  through  the  software.  If  the  teacher  select  the  "By  Performance"  option,  the 
software  will  determine  the  number  of  problems  he/she  must  complete  in  mastering  a  skill,  the 
number  of  helps  allowed  for  the  student,  and  a  range  of  acceptable  errors  and  steps  that  are 
deemed  acceptable  for  the  student's  current  level  of  performance. 
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APPENDIX  B:  MAESTRO:  THE  WRITING  PROCESS  TUTOR 


Goal  of  Tutor 

Maestro:  The  Writing  Process  Tutor  facilitates  the  development  of  basic  writing  process  skills. 
The  tutor  guides  students  through  prewriting,  drafting,  revision,  and  editing  phases  of  the  writing 
process,  while  helping  them  to  structure  writing  activities,  and  develop  the  cognitive  skills  used 
in  writing. 

Curriculum 

Writing  Process 


Prewriting.  While  working  in  this  stage,  the  writer  will  identify  goals,  analyze  the  audience, 
decide  the  style  and  purpose,  and  develop  a  thesis.  Students  will  also  generate  ideas  using  one  of 
the  four  idea  tools:  clustering,  outlining,  note  taking,  and  cubing.  Students  also  construct  a 
writing  plan  during  their  pre writing  activities. 

Composing  a  draft.  Draft  helps  writers  use  their  ideas  to  compose  well- written  sentences  and 
form  effective  paragraphs. 

Editing.  Writers  are  first  guided  through  the  revising  process  by  selecting  appropriate  questions 
to  apply  to  the  composition.  Students  using  the  tutor  revise  on  different  levels:  Whole  Paper, 
Introduction,  Body,  Conclusion,  and  Sentence. 

Publish  allows  the  writer  to  tailor  the  written  presentation  to  the  audience.  It  provides 
workspaces  for  proofreading,  formatting,  highlighting,  and  distributing  their  work. 

Types  of  Writing 

Maestro  supports  most  writing  purposes  including  expository,  persuasive,  research,  and  practical. 
Teachers  can  choose  the  type  of  writing  required  of  the  students  and  the  specific  challenging 
assignments  presented  by  Maestro. 

Expository  Writing.  Expository  essays  explore  and  present  information  to  a  reader.  Students  are 
guided  throughout  the  development  of  a  multi- paragraph  composition.  Most  of  these  essays  will 
entail  a  process  or  incident. 

Persuasive  Writing.  The  purpose  of  a  persuasive  essay  is  to  influence  or  change  a  person's 
opinion  about  a  specific  topic.  In  Maestro,  persuasive  essays  may  take  the  form  of  editorials, 
letters,  or  opinion  pieces. 

Research  Writing.  Research  writing  requires  a  well- documented  and  carefully  planned  report. 
These  research  papers  may  be  either  informative  or  analytical. 
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Practical  Writing.  Practical  writing  prepares  students  to  structure  their  writing  to  achieve  joint 
understanding  with  their  readers.  This  aspect  of  the  tutor  is  designed  to  be  relevant  in  today's 
business  environment:  resume,  cover  letter,  business  letter,  memo,  and  e-mail.  Students  are 
guided  through  the  actual  writing  process  and  provided  with  feedback  to  ensure  that  their  work 
meets  quality  standards  of  the  business  world. 

Intended  Audience 

Maestro  was  initially  developed  for  ninth-grade  students,  but  has  been  extended  to  middle  school 
and  young  adult  populations,  such  as  those  attempting  to  complete  a  GED. 

Instructional  Features 

Tailored  instructional  modules  (TIMs).  TIMs  are  short,  simple  instruction  focused  on  the 
specific  writing  needs  of  each  student.  TIMs  are  adaptive  to  the  interests  and  learning  styles  of 
individual  students  for  each  stage  of  the  writing  process  including  goat  setting,  generating  and 
organizing  ideas,  generating  a  writing  plan,  drafting,  revising,  and  publishing. 

Student  Controlled  Workspaces.  Student- controlled  workspaces  designed  to  facilitate  the 
development  of  skills  associated  with  the  stages  of  the  writing  process.  The  workspaces  adapt 
based  on  the  type  of  writing  required  of  the  student. 

Challenging  Assignments.  Situated  assignments  help  motivate  students  by  using  multimedia- 
based  stories  and  realistic  writing  tasks. 

Intelligent  Advice.  An  intelligent  advisor  tutors  student  by  monitoring  student  progress, 
providing  diagnostic  advice,  presenting  appropriate  TIMs,  and  managing  workspaces. 

Interfaces 

There  are  over  20  different  interfaces  or  workspaces  for  the  students  to  use  while  developing 
writing  process  skills.  These  are  selected  and  presented  by  the  intelligent  controller  within 
Maestro. 

Teacher  Toolkit  Features 

The  design  of  Maestro  allows  it  to  be  integrated  into  any  writing  curriculum.  Teachers  manage 
the  rosters,  lessons,  writing  assignments,  reports,  and  testing. 

Rosters.  Teachers  enter  class  rosters,  display  rosters,  and  add,  delete,  or  update  student  records. 

Class  Lessons.  Teachers  construct  lessons  by  selecting  one  or  more  writing  assignments  for  the 
students  to  select. 

Writing  Assignments.  Teachers  can  author  new  writing  assignments  tailored  to  their  students, 
curriculum,  and  local  community.  Each  assignment  can  include  a  writing  prompt,  a  reading,  a 
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picture  or  graphic,  and  a  video.  Teachers  also  determine  the  number  of  paragraphs  the  students 
must  complete,  the  type  of  writing,  and  which  workspaces  are  required  or  available  for  the 
students  to  use. 

Reports.  Teachers  can  view  on  the  computer  screen  or  print  out  the  class  rosters,  students' 
profiles,  their  writing  in  each  and  every  workspace,  or  a  summary  of  the  students'  progress. 

Pretesting  and  Posttesting.  Teachers  can  give  an  automated  pretest  and  posttest  using  the 
teacher  tools. 
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APPENDIX  C:  INSTRUCTION  IN  SCIENTIFIC  INQUIRY  SKILLS  (ISIS) 

Goal  of  Tutor 

Instruction  in  Scientific  Inquiry  Skills  (ISIS)  tutor  focuses  on  developing  students'  critical 
thinking  skills,  scientific  literacy,  and  scientific  inquiry  skills  in  the  context  of  ecology  and  the 
life  sciences. 

A  goal  of  science  education  is  to  produce  students  who  are  competent  in  science.  By  "competent 
in  science,"  the  project  team  means  that  each  student  will  be  literate,  functional,  and  critical  in 
the  domain  of  science.  To  be  literate  denotes  the  abilities  to  obtain,  comprehend,  and 
communicate  scientific  material.  To  be  functional  refers  to  the  ability  to  utilize  the  methods, 
principles,  and  technologies  that  pertain  to  science.  To  be  critical  indicates  the  abilities  to  assess 
the  soundness  of  scientific  approaches  and  outcomes  and  to  judge  the  significance  of  science  and 
technology  in  society. 

Curriculum 

Students  are  required  "to  do"  science  in  ISIS.  They  first  pick  one  of  the  more  than  300  research 
questions  from  a  series  of  concept  maps.  The  concept  maps  are  organized  by  topic  areas,  such  as 
Human  Activities.  Once  they  have  selected  a  research  question,  they  perform  the  following 
skills  in  ISIS. 

Skills 


The  over- arching  goal  of  the  science  tutor,  Instruction  in  Scientific  Inquiry  Skills  (ISIS),  is  to 
increase  the  level  of  scientific  functioning  of  high  school  students  enrolled  in  Introductory 
Biology.  Because  this  level  of  functioning  is  too  broad  to  address  in  the  initial  design  of  ISIS, 
the  tutor  focuses  on  skills  underlying  scientific  inquiry.  Some  of  the  activities  required  in 
applying  scientific  methods  are  automated  within  ISIS;  other  skills  constitute  what  the  students 
will  learn  by  interacting  with  the  computer.  The  following  are  the  specific,  measurable 
objectives  of  ISIS.  Specifically,  students  will  be  able  to: 

•  Generate  a  research  question. 

•  State  a  testable  hypothesis. 

•  Design  a  controlled  experiment  to  test  that  hypothesis. 

•  Conduct  the  experiment  in  a  simulated  environment. 

•  State  a  conclusion  from  the  experiment. 

•  Accept  or  reject  hypothesis. 

Domain  Content 


These  scientific  inquiry  skills  are  taught  within  an  environmental  theme.  Students  learn  about 
ecological  concepts  by  selecting  and  completing  research  questions.  These  research  questions 
are  categorized  into  the  following  five  major  categories  and  their  respective  subcategories: 
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Biomes:  Grasslands,  Deserts,  Temperate  Deciduous  Forests,  Coniferous  Forests,  Tropical 
Rainforests,  Polar  Regions,  Tundra,  Fresh  Water,  Marine. 

Abiotic  Factors:  Atmosphere,  Water,  Sunlight,  Ph/Chemicals,  Rainfall,  Irradiance,  Soil,  Natural 
Disasters. 

Biotic  Factors:  Autotrophs,  Heterotrophs,  Decomposers. 

Ecology:  Energy  Flow,  Natural  Relationships,  Material  Recycling,  Genetic  Variation 

Human  Factors:  Atmospheric  Pollution,  Land  Use,  Natural  Resources,  Habitat,  Genetic 
Diversity,  Water  Pollution. 

Intended  Audience 

The  content  of  ISIS  in  which  inquiry  skills  are  taught  is  environmental  science.  As  with  the 
word  problem  solving  and  writing  process  tutors,  the  software  is  primarily  geared  toward  middle, 
junior  high,  and  high  school  populations. 

Instructional  Features 

Research  Assignments.  ISIS  includes  over  300  research  assignments  including  detailed 
information  about  the  assignment,  the  correct  independent  and  dependent  variables,  the  correct 
relationship  between  the  variables,  the  units  of  measure,  equipment  needed  for  each  assignment. 
These  are  used  by  ISIS  to  monitor  the  student's  progress  as  they  complete  a  research  assignment. 

Simulations.  ISIS  includes  five  biome  simulations  (e.g.,  coniferous  forest,  deciduous  forest, 
tundra,  grasslands,  desert).  When  conducting  an  experiment,  students  enter  a  simulation,  set  the 
independent  variable,  set  the  values  for  five  experimental  groups  (i.e.,  levels  of  the  experimental 
variable),  and  run  the  simulation  for  three  growing  seasons. 

Skill  Instructional  Modules.  The  instructional  modules  are  lessons  on  five  scientific  inquiry 
skills.  After  each  lesson,  a  student  progresses  to  the  castle  to  complete  assignments  using  the 
skills  he/she  has  just  learned.  These  skills  instructional  modules  include:  Generating  a 
Hypothesis,  Designing  an  Experiment,  Conducting  an  Experiment,  Drawing  a  Conclusion  and 
Accepting  or  Rejecting  a  Hypothesis. 

Generate  Hypothesis:  This  instructional  module  teaches  students  how  to  develop  a 
research  question  and  generate  a  hypothesis  by  introducing  several  subskills.  The  subskills 
taught  in  generating  a  good  hypothesis:  observing  graphs,  generating  observations  from  text, 
generating  questions  from  visuals,  questioning  graphs,  generating  questions  from  text,  generating 
hypothesis,  testability  of  hypothesis,  and  rating  scientific  inquiry  for  importance  and  testability. 

Designing  an  Experiment:  This  module  shows  students  how  to  specify  independent  and 
dependent  variables,  units  for  the  variables,  a  time  period  for  an  experiment,  and  equipment  used 
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in  an  experiment.  The  subskills  taught  in  designing  an  experiment  are:  identifying  variables, 
forming  test  groups,  measuring  units  and  equipment,  and  determining  time  span. 


Conducting  an  Experiment:  This  module  shows  a  student  how  to  properly  organize, 
analyze,  and  display  the  data  while  the  experiment  is  being  conducted.  The  subskills  are: 
executing  a  procedure,  collecting  data,  and  presenting  data. 

Drawing  a  Conclusion  and  Accepting  or  Rejecting  a  Hypothesis:  This  module  teaches  a 
student  how  to  effectively  interpret  the  data  and  compare  the  results  to  the  hypothesis  to  draw  a 
conclusion  and  accept  or  reject  the  hypothesis.  Since  this  module  is  relatively  short,  the  two 
skills  are  not  broken  into  subskills. 

Domain  Instructional  Modules.  The  domain  instructional  modules  contain  text  and  graphic 
information  on  ecology  or  biology  concepts  related  to  the  assignments  in  the  tutor.  The  student 
must  answer  questions  at  the  end  of  each  module  to  continue  in  the  tutor. 


Acid  Rain 

Resources 

Biomes: 

Agriculture 

Seasons 

Coastal  Ecosystems 

Atmosphere 

Soil 

Coniferous  Forests 

Biomass 

Solar  Radiation 

Deciduous  Forests 

Deforestation 

Succession 

Deserts 

Desertification 

Symbiosis 

Fresh  Water  Biome 

Ecosystem 

Temperature 

Rainforests 

Ecosystem  Development 

Tundra 

Food  Chain 

Polar  Regions 

Freon/Chlorofluorcarbons 

Chemicals: 

Greenhouse  Effect 

Carbon 

Human  Population 

Carbon  Dioxide 

Light 

Helium 

Movement  of  Energy  and 
Materials 

Hydrogen 

Natural  Disasters 

Nitrogen 

Photosynthesis 

Oxygen 

Plant  Structure 

Ozone 

Pollutants 

Water 

Pretest  and  Posttest.  ISIS  includes  two  pretests  and  posttests  which  the  teacher  can  schedule 
using  one  of  the  teacher  utilities.  One  pretest-posttest  combination  assesses  student 
understanding  and  application  of  the  scientific  inquiry  skills.  The  second  combination  is  focused 
on  the  biology,  ecology,  and  environmental  science  content. 

Interfaces 

The  interface  of  ISIS  is  unique.  Students  find  themselves  in  a  medieval  castle,  where  they  must 
click  on  various  objects  to  do  an  assignment.  The  castle  includes  a  multitude  of  characters  and 
functions: 
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Igor,  the  lab  assistant,  guides  students  and  provides  feedback  about  their  performance  on 
the  each  step  of  scientific  inquiry  process 

Wizard  gives  hints  about  what  the  student  is  doing  wrong  or  what  is  needed  to  complete  a 
part  of  the  assignment. 

Magic  Windows  contain  the  ecological  topics  (i.e.,  research  questions)  addressed  in  the 

tutor. 

Equipment  Room  contains  supplies  needed  to  conduct  the  experiment:  meters,  ppm 
meters,  linear/mass  devices,  glassware,  and  timers. 

Library  contains  the  Skill  and  Domain  Instructional  Modules,  a  glossary,  and  interface 

help. 

Glossary  contains  all  of  the  words  highlighted  in  the  domain  instructional  modules. 

Desk  is  where  the  students  do  their  work.  The  desk  and  the  area  around  it  include  a 
crystal  ball,  terrarium,  notebook,  conclusions  book,  final  decisions  book,  skull,  and  a  map. 
Students  make  predictions  (generate  a  hypothesis)  using  the  crystal  ball.  The  terrarium  is  used  to 
design  and  conduct  their  experiments.  The  conclusions  book  allows  the  student  to  draw  a 
conclusion  based  on  the  results  of  the  experiment  while  the  final  decisions  book  is  where  the 
student  ultimately  accepts  or  rejects  their  hypothesis. 

Toolbox  contains  the  student's  notebook  and  equipment  selected  in  the  equipment  room. 

Notebook  has  three  sections  regarding  their  assignments:  'Current  Work',  'Prior 
Work/Notes',  and  'Assignment  Information'. 

Treasure  Chest  is  where  the  students  "spend"  the  points  accumulated  by  successfully 
completing  the  assignments,  SIMs,  and  DIMs. 

Theoretical/Development  Background 

A  cognitive  apprenticeship  approach  to  instruction  is  embedded  in  ISIS.  Cognitive 
apprenticeship  emphasizes  two  relevant  issues.  First,  cognitive  apprenticeship  aims  to  teach 
cognitive  and  metacognitive  skills  used  by  experts  when  faced  with  complex  tasks  (Collins, 
Brown,  &  Newman,  1989;  Collins,  Hawkins,  &  Carver,  1991).  Cognitive  apprenticeship 
approaches  focus  on  teaching  students  how  to  solve  complex  problems  and  tasks.  This  requires 
the  instructor  to  teach  declarative,  procedural,  and  metacognitive  knowledge  in  the  context  in 
which  it  is  used. 

The  second  issue  addressed  by  cognitive  apprenticeship  is  the  actual  method  of  teaching 
students.  Cognitive  apprenticeship  relies  on  students  learning  through  guided  experiences 
(Collins,  et  al,  1989).  Initially,  an  instructor  models  the  external  behaviors  while  describing  the 
cognitive  aspects  underlying  the  performance  of  the  task- related  skills.  Following  the  modeling 
of  the  skills,  the  student  performs  those  skills  under  the  direction  of  the  instructor.  The  instructor 
guides  the  student  through  the  task  providing  lots  of  structure  or  guidance  to  the  student.  As  the 
student  becomes  more  proficient  in  performing  the  task,  the  instructor  "fades"  or  reduces  the 
amount  of  structure.  Throughout  the  teaching- learning  episode,  the  student  is  required  to  reflect 
upon  his/her  performance  relative  to  that  of  experts.  This  reflection  develops  self-correction  and 
self- monitoring  skills.  The  methods  underlying  a  cognitive  apprenticeship  approach  to 
instruction  (Collins,  et  al.,  1989)  include;  (a)  modeling  of  the  cognitive  skills  and  processes 
underlying  task  performance,  (b)  coaching  the  student  during  his/her  performance  of  the  skills, 

(c)  structuring  of  lesson  content  and  subsequent  fading  of  the  structure,  (d)  requiring  the  student 
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to  reflect  upon  his/her  performance  and  articulate  his/her  knowledge,  and  (e)  allowing  students  to 
explore  how  to  apply  newly  learned  information  in  accomplishing  complex  tasks. 

For  ISIS  to  accomplish  the  methods  underlying  the  cognitive  apprenticeship  approach,  the  tutor 
must  be  able  to  perform  several  instructional  activities.  The  set  of  activities  chosen  to  implement 
the  cognitive  apprenticeship  approach  is  described  by  Hsieh,  Miller,  Hicks,  and  Lorenz  (1993). 
The  tutor  must  be  able  to  (a)  present  detailed  domain  knowledge  and  inquiry  skills  and  multiple 
examples  of  each,  (b)  elicit  performance  from  the  student,  (c)  diagnose  the  student's 
performance,  and  (d)  give  appropriate  feedback. 

One  component  of  instruction  in  the  cognitive  apprenticeship  approach  is  to  allow  the  students  to 
explore  how  to  apply  their  knowledge  in  accomplishing  complex  tasks  (Collins,  et  al.,  1989).  In 
addition,  Project  2061  states  that  "students  need  time  for  exploring,  for  making  observations,  for 
taking  wrong  turns,  for  testing  ideas,  for  doing  things  over  again...”  (Rutherford  &  Ahlgren, 
1990,  pi 93).  Allowing  students  to  "play"  in  a  computer-based  simulation  of  the  domain 
concepts  is  one  way  to  achieve  these  recommendations.  ISIS  contains  a  simulation  of  simple 
ecosystems,  such  as  a  grassland  and  a  coniferous  forest,  so  that  students  can  conduct  experiments 
testing  their  hypotheses. 

The  role  of  a  student  model  is  to  provide  information  to  the  instructional  module  to  make 
instructional  decisions  (Self,  1990;  Van  Lehn,  1988).  The  student  model  is  a  repository  of  the 
interpretations  made  by  the  instructional  module  during  the  diagnosis  of  the  student's  behavior. 
The  student  model  needs  to  record  information  for  immediate,  on-line,  instructional  and  long¬ 
term,  deferred,  curricular  decision  making.  The  immediate  decisions  tend  to  revolve  around  the 
instructional  action  the  tutor  must  take  given  a  student's  behavior.  The  tutor  diagnoses  the 
student's  performance  and  then  must  determine  whether  to  give  any  feedback  and  what  kind. 

The  long-term,  deferred  decision-making  primarily  involves  decisions  at  a  curricular  level.  Once 
a  student  has  completed  an  assignment,  the  tutor  needs  to  determine  whether  to  advance  a 
student  to  another  unit  of  instruction  or  to  assign  a  task  within  the  same  instructional  unit. 

The  types  of  data  recorded  in  a  student  model  includes  data  about  a  student's  progress  through 
the  curriculum,  the  student's  knowledge  of  the  biology  concepts,  and  the  student's  ability  to 
perform  the  skills  underlying  the  scientific  method.  The  curricular  information  includes  the  list 
of  assignments  completed,  the  current  assignment,  and  status  on  the  current  assignment.  Data  on 
the  biology  concepts  include  whether  a  student  has  received  instruction  on  each  concept,  the 
number  of  errors  a  student  has  made  on  the  current  assignment,  the  number  of  errors  the  student 
has  made  on  previous  assignments  involving  the  concept,  and  a  belief  about  the  student's 
proficiency  based  on  these  and  other  data.  Data  on  the  scientific  skills  include  whether  a  student 
has  received  instruction  on  each  skill,  the  number  of  errors  a  student  has  made  on  the  current 
assignment,  the  number  of  errors  the  student  has  made  on  previous  assignments  involving  the 
skill,  and  a  belief  about  the  student's  proficiency  based  on  these  and  other  data. 
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Evaluation 


A  pilot  evaluation  of  this  tutor  was  conducted  during  the  1994-1995  academic  year  followed  by 
large-scale  field  evaluations  in  1995-1996,  1996-1997,  1997-1998,  and  1998-1999.  Generally, 
students  show  an  11%  increase  in  their  scientific  inquiry  skills  after  having  used  ISIS  for  15  or 
more  hours.  These  studies  and  results  are  summarized  in  the  FST  Evaluation  Summary  and 
several  research  papers. 

Teacher  Toolkit  Features 

The  ISIS  Teacher  Toolkit  allows  teachers  to  keep  track  of  their  students'  work  electronically. 

Rosters.  Using  the  Teacher  Toolkit,  teachers  can  enter  student  rosters,  quickly  find  a  specific 
user,  remove  a  student,  sort  the  rosters  by  different  variables,  verify  login  information,  and  edit 
user's  information. 

Generating  and  Printing  Reports.  Several  types  of  reports  may  be  generated.  Types  of  reports 
include:  Skill  Proficiency  Summary,  Domain  Reading  Summary,  Notebook,  Class  Roster,  and 
Student  ID  Cards. 

The  Skill  Proficiency  Summary.  This  report  is  a  detailed  presentation  of  each  student's 
performance  on  each  of  the  scientific  inquiry  skills.  It  includes  the  number  of  assignments 
students  have  completed  for  each  skill  level,  the  number  of  errors  students  have  made,  and  the 
number  of  help  statements  students  have  requested.  Overall,  proficiency  is  calculated  using  all 
completed  problems. 

The  Domain  Reading  Summary.  The  Domain  Instructional  Module  Summary  produces 
information  on  the  students'  interactions  with  the  domain  instructional  modules.  It  presents  the 
number  and  amount  of  time  each  student  sent  in  each  DIM. 

Notebook.  Teachers  can  print  out  the  students  notes  and  research  outcomes  on  each 
assignment.  This  is  a  great  tool  for  portfolio  assessments. 

The  Class  Roster  lists  of  all  the  students  in  a  particular  class  period.  The  Student  ID 
Cards  prints  labels  for  cards  students  can  use  to  log  into  the  software.  The  labels  contain  teacher 
name,  period,  student  name,  user  ID,  and  password. 

ISIS  Configuration  Utility  and  Test  Utility.  Instructors  may  enable  or  disable  the  science  tests 
using  this  utility.  The  software  may  be  set  to  give  no  test,  give  a  content  or  skills  pretest  the  first 
time  a  user  logs  on  or  give  a  matching  content  or  skills  posttest  the  next  time  a  user  logs  on.  The 
Test  Utility  allows  teachers  to  view  student  test  scores  at  anytime.  The  user  may  also  save  the 
results  to  a  file  or  print  it. 
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APPENDIX  D:  SITES  PARTICIPATING  IN  THE  FST  PROJECT 


ALBUQUERQUE  AND  NM 

Bernalillo  High  School 
Los  Lunas  High  School 
Cuba  Middle  School 
Manzano  High  School 
Mesa  Alta  Junior  High  School 
Silver  High  School 
Socorro  High  School 
West  Mesa  High  School 
La  Cueva  High  School 
Dora  High  School 
Gadsden  High  School 
Grants  High  School 
Hagerman  High  School 
Hatch  High  School 
Hondo  High  School 
Maxwell  High  School 
Moriarity  High  School 
Onate  High  School 
Tatum  High  School 

SAN  ANTONIO.  TX 

Mac  Arthur  High  School 
Sam  Houston  High  School 
Harlandale  Academy 
Gamer  Middle  School 
Krueger  Middle  School 
Roosevelt  High  School 
Harlandale  High  School 
McCollum  High  School 
Harlandale  Alt  Center 
South  San  High  School 
West  Campus  High  School 
Woodlake  Middle  School 
Judson  High  School  -  Gray 
St  Matthew  Catholic  School 


DAYTON,  OH 

Trotwood- Madison  High  School 
Dunbar  High  School 

ALLENTOWN,  PA 

Salisbury  Middle  School 

ROME,  NY 

Staley  Middle  School 
Strough  Middle  School 
Mohawk  Valley  CC 


DEPARTMENT  OL  LABOR 

Penobscot  Job  Corp  Center 
Loring  Job  Corp  Center 
Westover  Job  Corp  Center 
Ouachita  Job  Corp  Center 
Dayton  Job  Corp  Center 
Hubert  Humphrey  Job  Corp  Center 
Connecticut  Job  Corp  Center 
Atterbury  Job  Corp  Center 
Montgomery  Job  Corp  Center 
Bangor  Career  Advancement  Center 
Memphis  Youth  Fair  Chance 
Edinburgh  Youth  Fair  Chance 

Alaska  Job  Corp  Center 
Cascades  Job  Corp  Center 
Clearfield  Job  Corp  Center 
Hawaii  Job  Corp  Center 
Northlands  Job  Corp  Center 
Phoenix  Job  Corp  Center 
Shriver  Job  Corp  Center 
South  Bronx  Job  Corp  Center 
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APPENDIX  E:  EVALUATION  SUMMARY 


1.  Word  Problem  Solving  (WPS)  Tutor  Pilot  Study 

Dates:  Sep  91  -  May  92 

Description:  A  pilot  test  was  conducted  to  investigate  the  instructional  effectiveness  of  the  first  version  of  WPS 
and  to  examine  issues  surrounding  a  long-term  operational  implementation  of  WPS  in  a  high  school  setting.  The 
treatment  group  consisted  of  400  students  in  16  sections  of  pre -algebra  with  eight  teachers  for  approximately  20 
hours  of  instruction.  A  demographically  similar  high  school  served  as  the  non -treatment  control  group. 

Outcome  The  tutored  group  significantly  outperformed  the  control  group  on  structured  word  problem  solving 
exercises.  In  addition,  their  knowledge  of  the  steps  involved  in  solving  word  problems  was  significantly  better  than 
the  control  group.  Several  issues  concerning  implementation  were  raised,  such  as  teacher-student-computer 
interaction  and  technical  support  requirements.  Data  from  this  study  were  applied  to  subsequent  tutor  design  and 
implementation. 


2.  Word  Problem  Solving  Tutor  Large-scale  Implementation  Study 

Dates:  Oct  92  -  May  93 

Description:  A  large-scale  evaluation  of  WPS  was  conducted  at  seven  schools  in  three  states  across  the  nation.  The 
treatment  group  used  the  WPS  software  plus  classroom  instruction,  the  placebo  group  used  the  same  set  of 
computerized  word  problems  but  with  no  theory -driven  instructional  approach  plus  classroom  instruction,  and  the 
control  group  used  traditional  classroom  instmction  only.  The  study  involved  632  students  and  20  teachers 
(Wheeler  &  Regian,  1999). 

Outcome :  Students  who  used  WPS  outperformed  students  who  used  the  placebo  tutor  and  students  who  received 
traditional  classroom  only.  The  treatment  group  improved  by  29%,  while  the  placebo  group  improved  by  18%  and 
the  control  group  improved  by  19%.  Further  implementation  issues  dealing  with  site  communication  were  raised 
and  solved.  In  addition,  differences  in  improvement  between  students’  concrete  reasoning  skills  and  abstract 
reasoning  skills  were  examined.  In  all  three  groups,  there  were  larger  increases  in  concrete  reasoning  performance 
than  in  abstract  reasoning  performance  (see  Table  1).  The  differences  in  change  scores  among  the  groups  on  both 
concrete  and  abstract  reasoning  scores  were  significant. 

Table  1.  Percent  Changes  in  Concrete  Reasoning  Skills  and  Abstract  Reasoning  Skills. 


Groups 

Percent  change  on 
concrete  reasoning 
scores 

Percent  change  on 
abstract  reasoning 
scores 

WPS 

31% 

20% 

Placebo 

19% 

15% 

Control 

22% 

11% 
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WPS 

Performance  Gains  By  Condition 

1  9  9  2  -  1  9  9  3 


3  0  % 


WPS  WPS 

Passive  Active  Control 

n  =  8  4  n  =  1  3  9  n  =  409 


3.  Reading  and  Writing  in  a  Supportive  Environment  (R-WISE)  Pilot  Study 

Dates:  Jan  93-  May  93 

Description:  The  pilot  test  was  conducted  to  investigate  the  effectiveness  of  the  first  version  of  R-WISE  and  issues 
surrounding  a  long-term  operational  implementation  of  it  in  a  high  school  setting.  The  treatment  group  consisted  of 
430  students  in  26  sections  of  ninth  grade  English,  nine  teachers,  approximately  9  hours  of  instruction.  Another 
demographically  similar  high  school  group  (423  students,  22  sections,  6  teachers)  in  San  Antonio  served  as  the  non- 
treatment  control  (Rowley,  Miller,  &  Carlson,  1997). 

Outcome :  The  treatment  group  had  higher  scores  on  five  measures  of  writing  performance  even  after  only  nine 
contact  hours  with  R-WISE.  Implementation  issues,  such  as  scheduling  classes  into  the  computer  laboratories,  were 
raised  and  resolved. 


Table  2.  R-WISE  Holistic  Pretest  Scores  and  Posttest  Results. 
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30% 
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-46% 

44% 
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R-WISE 

Purpose  Test  Scores 
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4.  A  Test  of  Two  WPS  Instructional  Approaches 

Dates:  Sep  93  -  May  94 

Description:  A  study  of  two  instructional  approaches  (guided  and  unguided)  was  conducted  at  two  schools  in 
Dayton,  OH  involving  194  students  and  5  teachers. 


Outcome :  A  significant  aptitude -treatment  interaction  was  found.  Students  with  lower  levels  of  achievement  as 
determined  by  their  class  level  performed  better  in  the  guided  mode  of  instruction  than  those  in  the  unguided  mode. 
Students  with  moderate  to  high  achievement  levels  performed  better  in  the  unguided  mode  than  the  guided  mode. 

WPS 

Performance  Gains  By  Condition 

1993-1994 


Student  Aptitude  and  Achievement 


Guided  mode  is 
better  for  students 
with  lower  aptitude 
and  achievement. 
Unguided  mode  is 
better  for  students 
with  higher  aptitude 
and  achievement. 


■  Guided 
— *■  -U  ngu  ided 
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5.  Reading  and  Writing  in  a  Supportive  Environment  (R-WISE)  Large-scale  Implementation  Study 
Dates:  Sep  93  -  May  94 

Description:  A  large-scale  operational  test  of  R-WISE  was  conducted  in  eight  schools  in  five  states  across  the 
nation.  The  treatment  group  (N=700  students,  17  teachers,  57  sections)  used  R-WISE  for  an  average  of  20  hours 
during  the  academic  year.  The  control  group  consisted  of  451  students  and  13  teachers  (37  sections)  from  the  same 
high  schools.  The  control  group  used  a  word  processor  for  an  average  of  20  hours  during  the  academic  year 
(Rowley,  Miller,  &  Carlson,  1997). 

Outcome :  The  students  using  R-WISE  significantly  outperformed  students  learning  to  write  using  a  word  processor 
on  all  five  measures  of  writing  performance.  The  attitudes  towards  writing  of  students  using  R-WISE  were 
significantly  more  positive  than  those  using  a  word  processor. 


R  -W  IS  E 


Performance  Gains  By  Condition 

1993-1994 
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6.  Extended  Operational  Test  of  the  Word  Problem  Solving  Tutor 

Dates:  Sep  94  -  May  95 

Description:  A  large-scale  field  test  of  a  revised  version  of  WPS  (v4.06)  was  conducted  with  611  subjects  at  eight 
schools  in  two  states.  In  order  to  test  the  effects  on  teacher  training  on  WPS  implementation,  training  was  shortened 
to  1  1/2  days  from  2  1/2  days. 

Outcome :  There  was  no  control  group  during  this  year.  Students  improved  their  word  problem-solving 
performance  by  12%.  Shortening  teacher  training  did  not  appear  to  negatively  affect  teacher  and  student  use  of 
WPS. 
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WPS 

Pre  -  and  Post-Test  Scores 
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7.  A  Comparison  of  Two  Instructional  Approaches  for  R-WISE 
Dates:  Sep  94  -  May  95 

Description:  Data  were  collected  from  students  using  two  versions  of  R-WISE.  One  version  had  extended, 
elaborated  instructional  modules,  which  the  students  were  required  to  complete  as  they  interacted  with  R-WISE. 

The  other  version  included  shorter,  simpler  instructional  modules.  The  relationship  between  teacher  style  and  tutor 
approach  was  investigated  (Rowley,  Miller,  &  Carlson,  1997). 

Outcome :  Data  from  the  first  semester  of  the  academic  year  showed  positive  gains  from  September  through 
January,  though  these  gains  were  smaller  than  previous  studies,  probably  because  student  contact  time  with  R-WISE 
was  significantly  reduced.  Due  to  a  network  installation  “bug”  in  the  updated  R-WISE  software,  teachers  used  the 
computer  lab  less  in  the  second  semester.  As  a  result,  the  data  from  the  spring  semester  are  inconclusive.  The  data 
on  teacher  style  interactions  indicate  that  styles  of  teaching  with  emphasis  on  student -centered  instruction  are  more 
compatible  with  the  tutor  produce  higher  student  scores.  However,  further  research  is  needed  before  strong 
conclusions  are  made  concerning  the  relationship  of  teacher  style  and  tutoring  approach. 
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8.  Instruction  in  Scientific  Inquiry  Skills  Tutor  (ISIS)  Pilot  Study 
Dates:  Oct  94  -  May  95 

Description:  A  pilot  test  was  conducted  to  investigate  the  effectiveness  of  the  first  version  of  ISIS  and  issues 
surrounding  a  long-term  operational  implementation  of  ISIS  in  a  high  school  setting.  The  treatment  group  consisted 
of  approximately  400  students  in  16  sections  of  ninth  grade  biology  with  four  teachers  for  approximately  15  hours  of 
instruction.  Ten  other  sections  of  freshman  biology  at  the  same  high  school  served  as  the  control  sample. 

Outcome :  Students  using  ISIS  gained  more  scientific  inquiry  skills  during  the  academic  year  than  their  control 
counterparts.  Qualitative  data  from  teacher-student-computer  interactions  indicated  several  directions  for 
refinements  for  the  subsequent  version  of  ISIS. 

9.  Small  School  Implementation  of  WPS 

Dates:  Jan  96-  May  96 

Description:  WPS  4.06  was  implemented  in  10  rural  sites  in  New  Mexico.  Teacher  training  was  shortened  to  one 
day  in  order  to  test  the  effects  of  reduced  teacher  training  sessions  on  the  implementation  of  WPS.  We  are  also 
testing  the  instructional  effectiveness  of  an  updated  version  of  WPS  in  those  schools. 

Outcome :  Students  showed  gains  comparable  to  previous  years.  Students  improved  their  word  problem-solving 
performance  by  6%.  We  met  with  site  coordinators  from  the  rural  sites  to  collect  information  on  the  implementation 
of  WPS  in  their  schools.  Site  coordinators  from  schools  with  only  one  math  teacher  using  WPS  reported  that  their 
teachers  sometimes  needed  to  discuss  questions  with  other  teachers.  We  are  addressing  how  to  help  users  in  remote 
sites. 
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10.  Addressing  Replicability  and  Measurement  Issues  in  R-WISE  studies 
Dates:  Sep  95  -  May  96 

Description:  Data  were  collected  from  students  using  the  previous  year’s  version  of  R-WISE  in  a  replication  study. 
Research  design  also  tested  for  equivalence  of  two  presumably  parallel  writing  prompts.  The  study  included  356 
students  of  8  English  teachers  in  the  treatment  group  and  261  students  of  5  Englis  h  teachers  in  the  control  group, 
with  39  English  classes  participating  (Rowley,  Miller,  &  Carlson,  1997). 

Outcome :  Findings  supported  the  results  of  previous  studies.  Results  indicated  a  small  but  significant  effect  of  the 
R-WISE  condition  on  learning  outcomes.  Results  also  raised  questions  as  to  the  reliability  of  the  two  prompts 
certified  as  equivalent.  Field  input  regarding  lessons  learned  suggested  directions  for  the  design  of  a  follow-on 
writing  process  tutor.  Maestro. 
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11.  Instruction  in  Scientific  Inquiry  Skills  (ISIS)  Tutor  Large-scale  Implementation  Study 

Dates:  Sep  95  -  May  96 

Description:  A  large-scale  test  of  the  effectiveness  and  implementation  of  ISIS  was  conducted  in  14  schools  in  five 
states.  There  were  over  30  teachers  using  ISIS  in  83  sections  of  ninth-grade  biology.  There  were  46  sections  in  the 
same  high  schools  serving  as  non -treatment  control  groups.  We  also  investigated  the  feasibility  of  using  ISIS  in  7th 
grade  honors  biology  and  we  explored  several  research  questions  concerning  the  use  of  concept  mapping  in  a 
knowledge -rich  domain  (Steuck  &  Miller,  1997). 

Outcome :  Observing  the  overall  gains,  the  treatment  group  improved  by  8%,  while  the  control  group  improved  by 
only  4%.  Specifically,  the  skill  level  obtained  by  the  students  directly  influenced  the  amount  of  gain.  Students 
using  ISIS  (having  met  a  threshold  of  assignments  completed)  outperformed  control  group  students  on  scientific 
inquiry  skills  as  measures  of  domain  knowledge.  Teacher  and  student  focus  groups  were  used  to  improve  the  design 
and  implementation  of  ISIS. 
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ISIS 


Total  Test  Score  By  Skill  Level 

1995-1996 


Skill  Level 


The  scores  of  the  design  experiment  subscale  demonstrate  the  fact  that  students  who  used  ISIS  and  obtained  higher 
skill  levels  outperformed  control  group  students  on  scientific  inquiry  skills  as  measures  of  domain  knowledge. 
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12.  A  Study  of  the  Effectiveness  of  the  Commercial  Version  of  WPS 

Dates:  Sep  96  -  May  97 

Description:  The  commercial  version  of  WPS  was  fielded  in  two  middle  schools  in  San  Antonio  and  three  high 
schools  in  New  Mexico  to  examine  the  effectiveness  of  WPS  with  younger  student  populations  and  to  replicate 
previous  results. 

Outcome :  Students  showed  gains  comparable  to  previous  years.  Overall,  students  (n  =  244)  improved  their  word 
problem-solving  performance  by  9%.  Specifically,  upward  gains  for  the  middle  schools  (n  =  63)  were  14%  while 
the  high  schools  (n  =  181)  showed  an  11%  upward  gain. 
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13.  Maestro:  The  Writing  Process  Tutor  Initial  Functionality  Study 

Dates:  Sep  96  -  May  97 

Description:  Maestro  was  designed  as  a  follow-on  tutor  to  R-WISE,  based  on  results  of  the  R-WISE  studies, 
teacher  input,  cognitive  research  into  the  writing  process,  and  the  cognitive  apprenticeship  instructional  strategy. 
The  functionality  and  efficacy  of  the  initial  version  of  Maestro  was  tested  in  a  large-scale  pilot  study.  The  study 
groups  include  classes  of  54  teachers  at  23  schools. 

Outcome :  Similar  to  results  with  the  other  tutors  in  the  FST  suite,  student  gains  were  directly  influenced  by  the 
amount  of  time  spent  on  the  Writing  Process  Tutor.  Specifically,  students  spending  at  least  1 1  hours  using  Maestro 
improved  by  1 1%  while  the  control  group  improved  by  only  3%. 

MAESTRO 

Performance  Gains  by  Use 
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14.  ISIS:  A  Replication  Study 

Dates:  Sep  96  -  May  97 

Description:  The  purpose  of  this  study  was  to  replicate  previous  findings  with  the  newest  version  of  the  software  in 
19  schools  with  over  30  teachers  utilizing  ISIS  (Meyer,  Steuck,  Miller,  Pesthy,  &  Kretschmer,  1999). 

Outcome:  Observing  the  overall  gains,  the  treatment  group  improved  by  11%,  while  the  control  group  improved  by 
only  6%.  Specifically,  the  skill  level  obtained  by  the  students  directly  influenced  the  amount  of  gain.  Once  again, 
students  using  ISIS  (having  met  a  threshold  of  assignments  completed)  outperformed  control  group  students  on 
scientific  inquiry  skills  as  measures  of  domain  knowledge.  Teacher  and  student  focus  groups  were  again  used  to 
improve  the  design  and  implementation  of  the  next  iteration  of  the  tutor. 


□  Posttest  Gain 

□  Pretest  Score 
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Pa  i  cent  Scors 
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As  seen  in  the  past  study,  the  design  subscale  showed  that  students  who  used  ISIS  and  obtained  higher  skill 
levels  outperformed  control  group  students  on  scientific  inquiry  skills  as  measures  of  domain  knowledge. 
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15.  A  Replication  Study  of  WPS  When  External  Requirements  Are  Decreased 

Dates:  Sep  97  -  May  98 

Description :  This  study  looked  at  the  effectiveness  of  the  software  when  external  requirements  were  decreased  for 
the  school  staff.  During  the  school  year,  the  technical  support  remained  readily  available  and  site  coordinator 
meetings  were  held.  However,  teachers  and  site  coordinators  were  not  required  to  complete  monthly  progress 
reports  or  participate  in  weekly  meetings  and  the  project  staff  eliminated  site  visits. 

Outcome :  The  data  were  collected,  but  not  analyzed  due  to  project  downsizing. 


□  Posttest  Gain 
b  Pretest  Score 


42 


16.  Maestro:  A  Replication  Study 

Dates:  Sep  97  -  May  98 

Description :  The  purpose  of  this  study  was  to  replicate  previous  findings  with  the  newest  version  of  the  software  in 
19  schools  with  over  30  teachers  utilizing  Maestro. 

Outcome :  The  data  were  collected  but  not  analyzed  due  to  project  downsizing. 

17.  ISIS:  A  Replication  Study 

Dates:  Sep  97  -  May  98 

Description:  The  purpose  of  this  study  was  to  replicate  previous  findings  with  the  newest  version  of  the  software  in 
four  schools  with  nine  teachers  utilizing  ISIS  (Meyer,  Miller,  Steuck,  &  Kretschmer,  1999). 

Outcome :  The  results  again  depicted  a  linear  trend  with  students  who  completed  more  assignments  showing  larger 
gains  on  both  a  Skills  and  Content  Test. 
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ISIS 

Scores  on  Design  Experiment  Subscale 
(Skills  Test)  By  Skill  Level 
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Consistent  with  the  earlier  ISIS  studies,  the  design  subscales  show  that  as  students  obtain  higher  skill  levels 
their  knowledge  is  increasing.  It  appears  that  there  exists  a  threshold  for  students  to  develop  a  coherent 
understanding  of  scientific  inquiry.  Students  may  have  to  progress  to  a  certain  level  of  experience  with  scientific 
inquiry  activities  in  order  to  get  “the  big  picture.”  This  is  supported  by  the  design  of  ISIS  in  that  students  reaching 
the  third  level  of  skill  development  use  the  simulated  biomes  to  conduct  their  experiments.  It  is  at  this  point  that 
they  carry  out  a  wide  range  of  scientific  activities:  planning  and  conducting  their  experiments. 
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